Scenario workflow based assessment system and method

ABSTRACT

A system and a method for automating performance assessments of an exercise or a training activity provide event assessment information in real-time to one or more evaluators in conjunction with unfolding events. The evaluators can wirelessly communicate assessment information to a database for after action review (AAR).

FIELD OF THE INVENTION

The invention pertains to systems and methods for assessing performanceof participants during training exercises, or, rehearsing of missions.More particularly, the invention pertains to automated systems andmethods to facilitate performance evaluation by providing real-timefeedback to evaluators as an activity proceeds.

BACKGROUND OF THE INVENTION

The importance of training personnel to respond to events such as fires,violent domestic events, accidents or natural disasters (earthquakes,tornadoes, floods or the like) is well recognized. Similar commentsapply to military training/mission rehearsal.

Training/rehearsal activities can last hours or days and can involve alarge number of geographically dispersed participants. The value ofcollecting information as to how the exercise was carried out tofacilitate an accurate and meaningful after-action review is also wellknown. One such system and method are disclosed in U.S. Pat. No.6,106,297 issued Aug. 22, 2000, assigned to the assignee hereof andentitled “Distributed Interactive Simulation Exercise Manager System andMethod”. The '297 patent is hereby incorporated by reference.

While the primary value of conducting a performance session, such as atraining or exercise session, is an effective and accurate assessment,(the basis of measurable and verifiable feedback to the session audienceor participant, the after-action review (AAR)) obtaining suchassessments during such sessions can be difficult. A problem inefficiently assessing, or evaluating, performance during complex tasksis defining what is important to be assessed at any given time, and whatassessment criteria should be used.

It has been known in prior art to define assessment criteria andguidance prior to the assessment session. The assessor is then requiredto monitor performance activities to determine what type of events aretaking place, recall and apply the applicable assessment criteria andassessment guidance, and record the applicable assessment. This approachis labor intensive, particularly for complex tasks involving teams ofseveral individuals, and teams in different locations.

There continues to be a need for improved, preferably real-timeevaluation systems and methods. Preferably such systems and methods willbe flexible and cost effective to implement so as to be usable toprovide assessment information for a wide range of civilian and militaryexercises.

SUMMARY OF THE INVENTION

A method which embodies the invention includes defining a scenarioworkflow relative to a selected situation, establishing a set ofassessment criteria relative to a plurality of workflow related events,carrying out the scenario; as the scenario proceeds, retrievingassessment criteria for at least one active event; and providing thereturned assessment criteria to an evaluator.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an over-view of a method in accordance with the invention;

FIG. 2 illustrates a system in accordance with the invention;

FIG. 3 illustrates an exemplary scenario event;

FIG. 4 illustrates an exemplary workflow event list for the event ofFIG. 3;

FIG. 5 illustrates an exemplary event assessment criteria list;

FIG. 6 illustrates a displayed event list for activating an event;

FIG. 7A illustrates an exemplary assessment entering screen; and

FIG. 7B illustrates an exemplary assessment prompting display for anactive event.

DETAILED DESCRIPTION OF THE EMBODIMENTS

While embodiments of this invention can take many different forms,specific embodiments thereof are shown in the drawings and will bedescribed herein in detail with the understanding that the presentdisclosure is to be considered as an exemplification of the principlesof the invention, and as a disclosure of the best mode of practicing theinvention. It is not intended to limit the invention to the specificembodiment illustrated.

Systems and methods that embody the invention improve the efficiency ofassessing performance during complex tasks, such as for distributedteams cooperating to achieve a common goal. Assessment accuracy isincreased while reducing the work associated with recording behaviorobservations, preparing material for briefings and debriefings, andpresenting feedback messages or comments during and after the assessmentsession.

Efficiency is increased because 1) the assessor is prompted during theassessment session with the applicable assessment criteria and guidance.An assessor can more quickly determine what assessment is needed; 2) theassessor is prompted with a tailored form for recording the assessment,so the assessor can record the assessment more quickly and accurately;3) assessment responsibilities can be efficiently distributed andallocated across several assessors during the assessment session, sofewer assessors are needed; 4) expert subjective judgment guidance thatis applicable to the assessment event can be captured from experts priorto the assessment session, and communicated to all assessors as requiredduring the assessment session, so the assessors are not required to beexperts.

Further, a method that embodies the invention automates performanceassessment of an activity such as a training or a rehearsal exercise(the assessment session). The automation results in measurable orobservable assessments of session events. In one embodiment, the methodincludes:

1) generating a scenario workflow which defines an event list or eventsthat are expected to occur during an assessment session;

2) generating a workflow event assessment list which defines assessmentcriteria or guidance to be used during a respective event;

3) defining active events from the event list;

4) prompting the assessor regarding the applicable assessment criteriarelative to a currently active event; and

5) recording the assessors' observations.

Alternative embodiments include:

a) Predefining the scenario workflow (e.g., a predefined script thatalways occurs in the same sequence).

b) The scenario workflow which defines an event list can be dynamicallyadjusted in response to the actions taken by the participants during theassessment session (e.g., a portion of the scenario workflow only occursif the participants behave in a certain way referred to as a triggeringbehavior).

c) The scenario workflow which defines an event list can be dynamicallyadjusted by the evaluator based on accomplishment of assessmentobjectives during the assessment session (e.g., the evaluator may addscenario workflow to increase the workload during an assessmentsession—referred to as an “inject”).

d) The workflow event list criteria can be predefined (e.g., thecriteria to be used for an event is always the same whenever that eventoccurs in the scenario workflow).

e) The workflow event list assessment criteria can be dynamicallyadjusted in response to the actions taken by the participants during theassessment session (e.g., if the participants or audience respond to ascenario situation).

f) The workflow event list assessment criteria can be dynamicallyadjusted by the evaluator based on accomplishment of assessmentobjectives during the assessment session (e.g., if the evaluator maydetermine that a report preparer has demonstrated mastery of the primarycriteria of submitting a properly formatted report in a timely fashion.The assessment criteria for future reporting could be adjusted to recordwhether the report preparer consults all relevant information sourcesneeded for a quality report).

g) Defining what event from the event list are currently applicable canbe based on human observation.

h) Defining what events from the event list are currently applicable canbe based on observation by a computer-based agent.

i) Prompting the assessor regarding the applicable assessment criteriacan be predefined (e.g., the assessor is always provided the same cue orinput screen each time the applicable event occurs).

j) Prompting the assessor regarding the applicable assessment criteriacan be dynamically adjusted in response to the actions taken by theaudience during the assessment session (e.g., if the participants arefrequently performing one type of task (Task A) and infrequentlyperforms another type of task (Task B), then the prompting means

k) Prompting the assessor regarding the applicable assessment criteriacan be dynamically adjusted by the evaluator based on accomplishment ofassessment objectives during the assessment session (e.g., if theevaluator determines that a report preparer has demonstrated mastery ofthe primary criteria of submitting a properly formatted report in atimely fashion. The assessment criteria for future reporting could beadjusted to record whether the report preparer consults all relevantinformation sources needed for a quality report).

In one system, that embodies this invention, the scenario workflow isdefined to detail sequences or series of events that would result for aparticular situation or course of action. The scenario workflow is thenstored in a scenario workflow event database. The scenario workflow isanalyzed to define what assessment criteria, if any, are applicable foreach of the workflow events. These assessment criteria are added to thescenario workflow event database.

During the assessment session, the assessor or evaluator is provided anautomated assessment device, or AAD. The AAD facilitates the use ofhand-carried Tablet PCs by the evaluation team to coordinate evaluationresponsibilities, display the pre-brief material and exercise status atany time during the exercise. As the scenario session proceeds, thescenario is monitored to determine what events are currently active.

By accessing the workflow event database, the system retrieves thedetails for the scenario event, including the assessment criteria listfor that event. The evaluator is prompted with context-sensitiveassessment criteria as key and critical events occur. As the evaluatorrecords observations against each exercise goal, AAD provides immediatefeedback of exercise goals that require additional attention. Thissignificantly reduces the need for the evaluator to closely monitoractual scenario event execution while recording key observations. TheAAD provides assistance to the evaluator to prepare the After ActionReview (AAR).

Following the exercise, the evaluation team can quickly review evaluatorcomments grouped by exercise objectives and select candidates fordiscussion. During the review, the evaluator can use the AAD to identifyand replay critical periods of audience actions that temporally relateto the evaluator's notes, while referencing the guidance provided duringthe exercise planning process. This automation and recall capability arenot available using current manual data collection and synthesistechniques. The resultant After Action Reports provide invaluablelessons for the exercise participants, exercise coordinators andstakeholder agencies, including best practices that can be shared, thus,enhancing emergency preparedness.

FIG. 1 illustrates an overall view of a method 100 in accordance withthe invention. In a step 102 a a scenario workflow is generated. In astep 102 b, a workflow event list is produced from the scenarioworkflow. The event list produced in step 102 b defines those eventswhich are expected to occur during a scenario assessment session.

In step 104 a, event assessment criteria are assigned. In step 104 b anevent assessment criteria list is generated. Those of skill willunderstand that the workflow event list step 102 b and event assessmentcriteria list step 104 b could be stored as known to those of skill inthe art for subsequent use.

In step 106 a the scenario is initiated and the participants or audienceparticipate in the ongoing scenario. Scenario workflow events unfold,step 106 b and the assessor 110 is available to observe theparticipants' or audience's performance in response to the events 106 b.

As the eventss unfold, step 106 b, active events are recognized, step108 either automatically or by personnel associated with implementingthe session.

Active events trigger a retrieval of evaluation or assessmentinformation, step 116. This information is forwarded to the assessor110, step 118.

The assessors' remarks, comments and evaluations are received step 120and stored for subsequent use, step 122 during after action review. Step118 can be repeated as appropriate to provide supplemental assessmentinformation to assessor 110 in view of previously recorded assessmentsstep 122.

FIG. 2 is a block diagram of a system 10 in accordance with theinvention. System 10 represents one embodiment for implementing themethodology 100 of FIG. 1. As illustrated in FIG. 1, the trainingaudience or participants 12 can communicate in a system 10 using, forexample, personal computers which communicate via an intranet 12 a andserver 12 b with the Internet 14. It will be understood that theparticipants 12 might participate in the subject scenario as multipleseparate teams or as one organization working together to respond to thescenario. The participants or audience 12 can receive information viatheir personal computers and intranet 12 a, such as by e-mail, telephoneor any other form of information providing systems, such as video,crisis information management system (CIMS) or the like.

The evaluators 110 can wirelessly communicate with a server associatedwith web 14 using wireless personal communication devices indicatedgenerally at 20. This wireless communication capability makes itpossible for the evaluators to readily move among the audience orparticipants during the exercise.

A plurality of web based servers 24 provides useful information for theexercise. An emergency scene simulator system 26 provides a realisticrepresentation of the subject scenario which requires the attention andaction of the training audience or participants 12. The e-mail, phone,crisis information management system 28 provides realistic communicationand coordination capabilities for the audience/participants 12. Thescenario log/replay system 30 records audience communications, scenarioevents, and emergency scene related simulation activity. System 32stores and makes available scenario events, procedures, standards,including the workflow event list, the event assessment criteria listsand related assessment information.

Role players 34 interact with the ongoing scenario and participants inthe session via one or more web based servers. They can communicate withthe audience or participants and/or control the sequence of events, theemergency scene simulator 26 and any other event related information.

As noted above, the scenario workflow is defined to detail sequences orseries of events that would result for a particular situation or courseof action, step 102 a. A sample scenario is illustrated in FIG. 3 (asingle event is shown for simplicity). The scenario events can then bestored in a database, such as database 32. It will be understood thatthe scenario events might be stored at a plurality of locations withoutdeparting from the spirit and scope of the invention.

For any given scenario event, there is a workflow procedure or eventsequence, the workflow event list step 102 b, that the audience isexpected to follow in response to the occurrence of the scenario events.The expected workflow event list 102 b for the sample scenario event inFIG. 3 is illustrated in FIG. 4. The scenario workflow event list 102 bcan then be stored in a database such as database 32, FIG. 2.

The scenario workflow is analyzed to define what assessment criteria, ifany, are applicable for each of the workflow events step 104 a. Theassessment criteria list, step 104 b, is created and stored in adatabase such as database 32. These criteria include a description ofthe objective behavior that is expected of the audience, an eventpriority used to designate relative priority of events, an EvaluatorObservation column to cue the assessor as to what should be observed,and a Performance Outcome note to define the criteria for successfulperformance by the participants or audience. These criteria which can beadded to the event list 102 b from FIG. 4, are illustrated in FIG. 5, asthe assessment criteria list step 104 b. The assessment criteria listcan be stored in a database such as 32.

Event determining software can be used to define what events from theevent list are currently active or applicable, step 108. As the scenariosession proceeds, the scenario is monitored to determine what events arecurrently active. This can be accomplished by monitoring the scenarioactivity and looking for a match of the expected “Trigger” condition forthose scenario events that are expected but have not yet occurred(status=“Pending”).

In one embodiment, active event identification can be accomplished by ahuman monitoring the event status see screen of FIG. 6, including alle-mail communication, as well as all voice between the role players andthe training audience. When the monitor determines that an event hadstarted, the monitor can designate that even as active (status=“Active”), using a PC-based workstation. This status of the events thatare currently applicable would be recorded, for example in database 32,and transmitted over the network to all role players 34 and evaluators110.

By assessing the workflow event database, the details for the activescenario event, including the assessment criteria list step 104 b forthat event can be retrieved, for example from database 32. The evaluator110 is prompted with context-sensitive assessment criteria as key andcritical events occur via an automated assessment device, or AAD 20.

The AAD 20, for example, hand-carried Tablet-PCs, enables the evaluationteam 110 to coordinate evaluation responsibilities, display lists onother exercise material and exercise status at any time during theexercise. FIGS. 7A and 7B are sample displays as might be presented tothe evaluation team 110.

FIG. 7A illustrates an exemplary normal AAD display where the scenarioevent monitor is shown in the bottom half of the display. The evaluatorrecords assessments in the top half of the display. As the evaluatorrecords observations against each exercise goal, the AAD providesimmediate feedback of exercise goals that require additional attentionstep 118. This significantly reduces the need for the evaluator toclosely monitor actual scenario event execution while recording keyobservations.

FIG. 7B illustrates AAD operation when a critical event has becomeactive. The evaluator is cued to this activity by the “alert” button,lower left comer. Upon selecting the alert button, an Alert Display thatis appropriate for assessing the current active event is displayed. Thisevent-appropriate cue enables extremely efficient assessment..

It will be understood that none of the details pertaining tocommunication via intranet 12 a or Internet 14 are limitations of theinvention. Similarly, those of skill in the art will recognize that noneof the size or location of the participants or audience 12, evaluator orassessment team(s) 110 or role players 34 are limitations of theinvention.

From the foregoing, it will be observed that numerous variations andmodifications may be effected without departing from the spirit andscope of the invention. It is to be understood that no limitation withrespect to the specific apparatus illustrated herein is intended orshould be inferred. It is, of course, intended to cover by the appendedclaims all such modifications as fall within the scope of the claims.

1. A method comprising: defining a scenario workflow relative to aselected situation; storing the workflow; establishing a set ofassessment criteria relative to a plurality of workflow related events;retrieving assessment criteria for at least one active event as thescenario proceeds; and providing the retrieved assessment criteria to anevaluator.
 2. A method as in claim 1 where the assessment criteria areprovided to the evaluator at or about the time the active event occurs.3. A method as in claim 2 where the criteria are communicated, at leastin part, wirelessly.
 4. A method as in claim 2 which includes storingreal-time evaluation information relative to the active event.
 5. Amethod as in claim 4 which includes providing supplemental informationrelative to evaluating the event, in response to previously storedobservations.
 6. A method as in claim 5 where the criteria arecommunicated, at least in part, wirelessly.
 7. A method as in claim 5where the supplemental information is provided in real-time as the eventis taking place.
 8. A method as in claim 7 which includes storingevaluation information during the event grouped at least bypredetermined objectives.
 9. A method as in claim 8 which includesreplaying stored information relative to the event, in combination withtemporally related, pre-stored, evaluation information.
 10. A method asin claim 9 which includes preparing reports pertaining to scenarioimplementation.
 11. A method as in claim 7 where the supplementalinformation is provided, in part, wirelessly to at least one evaluator.12. A method as in claim 7 which includes carrying out a plurality ofevents related to the scenario workflow.
 13. A method as in claim 12which includes wirelessly receiving assessment criteria related to aplurality of workflow related events.
 14. A method as in claim 13 whichincludes providing wireless receivers to the evaluators.
 15. A systemcomprising: a stored workflow event list; a stored event assessmentcriteria list; circuitry to provide information to an evaluator forassessing a plurality of events; and circuitry for providingsupplemental information to the evaluator.
 16. A system as in claim 15which includes circuitry to provide assessment criteria to a pluralityof evaluators.
 17. A system as in claim 16 which includes storage for aplurality of event assessments.
 18. A system as in claim 17 whichincludes a plurality of evaluator units which provide context-sensitiveassessment criteria.
 19. A system as in claim 18 where at least some ofthe units communicate wirelessly.
 20. A system as in claim 19 where theunits include software enabling an evaluator to enter exercise relatedobservations.
 21. A system as in claim 20 where the units includesoftware to receive and present feedback information to the evaluatorrelative to the exercise.
 22. A system for evaluating performance duringa scenario comprising: first software for creating a plurality ofscenario workflow events; second software for creating assessmentcriteria for each of the workflow events; software for determining atleast one active event of an on-going scenario; software for retrievingassessment criteria for the active event; software for receiving eventrelated assessment information from the evaluator; and softwareproviding for supplemental assessment information to an evaluator.
 23. Asystem as in claim 22 where the first software creates and stores atleast one workflow event list.
 24. A system as in claim 23 where thesecond software creates and stores a workflow event assessment criterialist.
 25. A system as in claim 22 which includes software for displayingassessment criteria for the active event.
 26. A system as in claim 22which includes software for displaying supplemental informationconcerning an on-going event to the evaluator.
 27. A system as in claim22 which includes wireless I/O devices for entering assessmentinformation.
 28. A system as in claim 19 which includes software forstoring assessment information.
 29. A system as in claim 28 forretrieving stored assessment information.